Multimodal sensor array for robotic systems

ABSTRACT

A multimodal sensing architecture utilizes an array of single sensor or multi-sensor groups (superpixels) to facilitate advanced object-manipulation and recognition tasks performed by mechanical end effectors in robotic systems. The single-sensors/superpixels are spatially arrayed over contact surfaces of the end effector fingers and include, e.g., pressure sensors and vibration sensors that facilitate the simultaneous detection of both static and dynamic events occurring on the end effector, and optionally include proximity sensors and/or temperature sensors. A readout circuit receives the sensor data from the superpixels and transmits the sensor data onto a shared sensor data bus. An optional multimodal control generator receives and processes the sensor data and generates multimodal control signals that cause the robot system&#39;s control circuit to adjust control operations performed by the end effector or other portions of the robot mechanism and when the sensor data indicates non-standard operating conditions.

RELATED APPLICATION

This application claims priority from U.S. Provisional PatentApplication No. 62/733,640, entitled “Multimodal Sensing ArchitectureFor Robotic Tactile Exploration”, filed on Sep. 20, 2018, U.S.Provisional Patent Application No. 62/733,641, entitled “Topology OfMultimodal Sensing With Integrated Readout Circuitry For Robots”, filedon Sep. 20, 2018, and U.S. Provisional Patent Application No.62/733,642, entitled “High-Speed, High-Bandwidth Multimodal TactileSensors On Bendable Platform”, filed on Sep. 20, 2018.

FIELD OF THE INVENTION

This invention relates generally to robotic systems and moreparticularly to sensors utilized to control robot mechanisms.

BACKGROUND OF THE INVENTION

Robotic systems typically integrate mechanical, electrical/electronicand computer science technologies in a way that produces autonomouslycontrolled mechanisms that selectively perform a variety of differentmechanical operations. For example, articulated robots are a class ofindustrial robotic systems in which an end effector (e.g., a hand orgripper) mounted on a robot “arm” mechanism is utilized to performrepetitive tasks, such as picking up a target object at one location andmoving the target object to a second location. The robot arm mechanismand end effector are generally controlled in accordance with aprogrammed series of movement operations that are based, for example, ona precise X-Y-Z starting location at which a target object will bereliably available for pick-up, and a precise X-Y-Z terminal location atwhich a receptacle is positioned to receive the target object whendropped off. While this programmed movement control approach isacceptable for use in highly ordered environments, erroneous andpossibly dangerous situations can occur when minor variations arise,such as displacement of a target object from the expected startinglocation or a receptacle is displaced at the terminal location, wherebyperformance of the programmed movement operations can result in damageto one or both of the target objects and the end effector/gripper. Toavoid such incidents, modern robotic systems often employ camera systemsand single-modal sensors (e.g., pressure sensors) that are mounted onthe end effector and provide feedback information that allows thesystem's control circuit to recognize and adjust for minor variations.

The lack of a rich end effector sensory feedback is one of the mainlimitations of modern robotic systems. That is, conventionalsingle-modality sensors (e.g., pressure sensing only) are unable toprovide sufficient information to avoid many common industrial accidentsand/or to perform complex assembly processes. For example, althoughsingle-modality pressure sensors provide sufficient data to verify thata predetermined gripping force is being applied by a hand-type endeffector onto a target object, they lack the rich sensor feedback neededto recognize when the target object is slipping from the end effector'sgrasp. In addition, when mounting a canister-type object over acylindrical object, single-modality pressure sensors provideinsufficient data regarding excessive contact between the cannister andcylindrical objects when the canister and cylindrical objects aremisaligned. Note that while camera-type feedback systems may be usefulto identify and adjust for such occurrences in some cases, criticalportions of the camera's field of view are often occluded by the endeffector, which limits the functionality of camera-type feedbacksystems. In contrast to single-modality sensors, the human hand consistsan unparalleled multimodal sensory system (i.e., mechanoreceptorssensing both pressure and vibration, and thermoreceptors sensingtemperature), which largely contributes to its unprecedented dexterousmanipulation. Specifically, the human multimodal sensing architectureprovides fine-grained cues about contact forces, textures, local shapearound contact points, and deformability, all of which are critical forevaluating an ongoing grasp, and to trigger force correction measures incase of instability.

What is needed is a sensing architecture for robotic systems thatovercomes the deficiencies of conventional single-modality sensors. Inparticular, what is needed is sensing architecture that mimicshuman-like tactile exploration to facilitate object-manipulation andrecognition tasks that present problems to robotic systems usingconventional single-modality sensors.

SUMMARY OF THE INVENTION

The present invention is generally directed to a multimodal sensingarchitecture that utilizes spatially arrayed multi-sensor groups(superpixels) to facilitate advanced object-manipulation and recognitiontasks performed by mechanical end effectors (e.g., a robot gripper/handattached to end of a robot arm mechanism) in robotic systems. In amanner similar to sensory receptors found in human fingers, thesuperpixels are spatially arrayed over contact surfaces of the endeffector (e.g., on the inward-facing surfaces of robot gripper fingers)such that each superpixel generates localized multimodal sensor data (e.g., data respectively generated by two or more different sensor types,or two or more types of sensor measurement) in response to stimuliapplied or received at an associated contact surface portion (i.e., theregion of the end effector's contact surface over which the superpixelis fixedly disposed). According to an aspect of the invention, eachsuperpixel includes at least one pressure sensor, at least one vibrationsensor, an optional proximity sensor and an optional temperature sensorthat collect corresponding sensor data in response to correspondingstimuli, thereby providing data that may be used to determineevents-of-interest occurring at each superpixel's associated contactsurface portion. The pressure sensor of each superpixel (e.g., a straingauge, a capacitive pressure sensor or a piezoelectric element) isconfigured to generate pressure (static event) data in response to anamount of static force applied to the corresponding surface portion, andthe vibration sensor of each superpixel (e.g., a piezoelectric sensor, apiezoresistive sensor or a MEMS accelerometer) is configured to generatevibration (dynamic event) data in response to mechanical vibrationsreceived at the corresponding surface portion. According to anotheraspect of the invention, a readout circuit receives the pressure dataand vibration data generated by the spatially arrayed superpixels andoperably transmits the received data to the robotic system's controlcircuit either directly (e.g., using a shared sensor data bus connectedbetween the readout circuit and the controller circuit) or indirectly(e.g., by way of an optional multimodal control generator that isconfigured to pre-process the “raw” sensor data before being passed tothe controller circuit). By providing the control circuit with bothstatic force and vibration data collected from the end effector in thismanner, the multimodal sensing architecture enhanced robotic systemcontrol based on both static events that occur on the end effector'scontact surface (e.g., the force by which an object is being gripped bythe end effector), and also dynamic events that periodically occur onthe end effector's contact surface (e.g., mechanical vibrationsgenerated when the object is slipping from the grasp of a gripper, ormechanical vibrations generated by contact between a grasped primaryobject and a secondary object). That is, by providing each superpixelwith both static and dynamic event data, the multimodal sensingarchitecture of the present invention greatly enhances a host roboticsystem's ability to quickly identify non-standard operating conditions(e.g., object slip or misaligned/misplaced objects) and automaticallyimplement a corrective operation (e.g., to adjust the gripping forceapplied by the end effector, or adjust the position of one objectrelative to an obstructive object). Even further enhancement of themultimodal sensing architecture's sensing capability may be achieved byway of utilizing proximity sensors in each superpixel to generateproximity data indicating distances between a target object and themultiple corresponding surface portions of the end effector, and/or byusing temperature sensors to generate temperature data indicating theamount of thermal energy transferred to multiple corresponding surfaceportions of the end effector. By forming superpixels that include allfour of these sensor types, the multimodal sensing architecture of thepresent invention enables robotic systems to utilize human-like tactileexploration (i.e., recognize vibrations, textures, and moments ofcontact with an object) to facilitate object-manipulation andrecognition tasks that greatly enhance the adaptability of robotmechanisms to a wide range of functional operations (e.g., automaticallyadjusting to random variations arising in repetitive tasks) that presentproblems to robotic systems using conventional single-modality sensors.

According to a practical embodiment of the present invention, a roboticsystem implements the multimodal sensing architecture by way ofdisposing two or more multimodal sensor arrays on associated contactsurfaces provided on opposing end effector fingers. In this case, eachmultimodal sensor array includes an associated feedback circuit that isoperably coupled to the robotic system's control circuit by way ofassociated sensor data buses that extend along the robot (arm)mechanism, where each feedback circuit is configured to receive sensordata from a large number of superpixels and to transmit the sensor datain a time multiplexed manner, whereby the large amount of sensor data isefficiently transmitted to the control circuit using a small number ofsignal lines. In one embodiment, the control circuit is customized toprocess the “raw” sensor data from the two or more multimodal sensorarrays that is transmitted on the sensor data buses. In anotherembodiment, the multimodal sensing architecture further includes amultimodal control generator that receives and processes the “raw”sensor data from one or more multimodal sensor arrays, and generatesmultimodal control signals that are then transmitted to the controlcircuit for use in controlling operations performed by the robotmechanism and the end effector.

According to another embodiment of the present invention, a method forcontrolling a robotic system involves utilizing one or more sensors togenerate both static event (e.g., pressure) data and dynamic event(e.g., vibration) data. As described above, the static event data isgenerated in response to static forces applied by a target object tocorresponding contact surface portions on the end effector whilegrasping the target object, and the dynamic event (e.g., vibration) datais generated in response to vibrational forces applied to thecorresponding contact surface portions on the end effector while therobot mechanism is being actuated to move the target object from onelocation to another location. The method also includes utilizing boththe static event data and the dynamic event data to identifynon-standard operating conditions while the target object is beingmoved, and adjusting the operation of the robot mechanism and/or the endeffector in response to the identified non-standard operating condition.For example, a combination of constant static event data and increasingdynamic event data is used to identify undesirable slipping of thetarget object due to insufficient friction between the end effector andthe target object. In this case, operation of the end effector isadjusted in response to the identified slipping condition, for example,by way of causing the end effector to increase the gripping forceapplied to the target object, thereby preventing undesirable dropping ofthe target object. In another example, a combination of constant staticevent data and sharply increasing dynamic event data is used to identifyundesirable impact-type or scraping-type contact between a transportedprimary object and a stationary secondary object resulting from anunscheduled misalignment of one or both objects. In this case, operationof the robot mechanism is adjusted in response to the identified contactcondition, for example, by way of translating (moving) the primaryobject in a way that removes the misalignment with the stationarysecondary object, whereby subsequent movement of the primary objectrelative to the secondary object produces acceptable sensor data. In apresently preferred embodiment, the static/dynamic event data isgenerated using the superpixel configuration described above. In analternative embodiment, the static/dynamic event data is generated usingan array of single multimodal sensors (e.g., piezoelectric sensors orpiezoresistive sensors) that are capable of detecting both static eventsand dynamic events. In this case, the associated readout circuit ismodified to include signal processing circuitry (e.g., filters, etc.)configured to separate static event characteristics from dynamic eventcharacteristics in each sensor's output signal, thereby enabling theabove-mentioned identification of non-standard operating conditions andassociated corrective adjustments using a smaller number of sensornodes.

According to another embodiment of the present invention, a topology ofthe multimodal sensing architecture includes embedded multi-node readoutcircuitry that is configured to extract sensor data generated by thesensor nodes (e.g., the pressure sensors and vibration sensors) of eachsuperpixel, and to coordinate periodic transmissions of the sensor datato the robotic system's control circuit over one or more shared signallines, thereby greatly simplifying the process of integrating multimodalsensing capabilities into existing robot systems by minimizing thenumber of signal lines. The topology consists of two main integratedparts: a multimodal sensing platform (sensor layer) and a custombackplane integrated silicon readout circuit (readout layer). The sensorlayer includes a silicon (or other) substrate upon which the superpixelsensor structures are fabricated using CMOS fabrication technologies orPCB fabrication processes. The readout layer includes an array ofaddressable readout circuit portions (pixels). In one embodiment eachreadout circuit portion includes a custom circuit that is capable ofboth reading voltage changes in analog sensor data signals andgenerating bias voltages or currents. In a specific embodiment, eachreadout circuit portion includes an analog front end with ananalog-to-digital converter (ADC) and a digital-analog-converter (DAC).Additionally, an optional protective matrix is formed over the sensorlayer, and an optional support substrate is disposed under the readoutlayer.

According to a presently preferred embodiment of the present invention,the multimodal sensing architecture is fabricated using a flexiblesubstrate material such that the sensor arrays can be flexed withoutsuffering any loss in performance. In a specific embodiment, the readoutcircuit of each array is fabricated using amorphous silicon (a-Si)thin-film transistor (TFT) elements.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other features, aspects and advantages of the presentinvention will become better understood with regard to the followingdescription, appended claims, and accompanying drawings, where:

FIG. 1 is a diagram depicting a multimodal sensing architectureimplemented on a robotic system according to an embodiment of thepresent invention;

FIG. 2 is simplified block diagram depicting the multimodal sensingarchitecture of FIG. 1;

FIGS. 3A, 3B, 3C and 3D are simplified side views depicting an operationperformed by the robotic system of FIG. 1 using static and dynamic eventdata generated by the multimodal sensing architecture of FIG. 1 inaccordance with an exemplary embodiment;

FIGS. 4A, 4B, 4C and 4D are simplified side views depicting an operationperformed by the robotic system of FIG. 1 using static and dynamic eventdata generated by the multimodal sensing architecture of FIG. 1 inaccordance with another exemplary embodiment;

FIG. 5 is a simplified cross-sectional side view showing a sensor arrayof a multimodal sensing architecture according to a specific embodiment;

FIG. 6 is an exploded perspective view showing an exemplary sensor arrayof a multimodal sensing architecture according to another specificembodiment;

FIG. 7 is a simplified cross-sectional side view showing a sensor arrayof a multimodal sensing architecture according to another specificembodiment;

FIG. 8 is a block diagram depicting a readout circuit of a sensor arrayof a multimodal sensing architecture according to another specificembodiment;

FIG. 9 is a simplified circuit diagram depicting a partial readoutcircuit of a sensor array of a multimodal sensing architecture accordingto another specific embodiment; and

FIG. 10 is a simplified cross-sectional side view showing a sensor arrayof a multimodal sensing architecture according to another specificembodiment.

DETAILED DESCRIPTION OF THE DRAWINGS

The present invention relates to an improvement in sensing architecturesutilized in robotic systems. The following description is presented toenable one of ordinary skill in the art to make and use the invention asprovided in the context of a particular application and itsrequirements. As used herein, directional terms such as “upper”,“lower”, “lowered”, “front” and “back”, are intended to provide relativepositions for purposes of description and are not intended to designatean absolute frame of reference. With reference to electrical connectionsbetween circuit elements, the terms “coupled” and “connected”, which areutilized herein, are defined as follows. The term “connected” is used todescribe a direct connection between two circuit elements, for example,by way of a metal line formed in accordance with normal integratedcircuit fabrication techniques. In contrast, the term “coupled” is usedto describe either a direct connection or an indirect connection betweentwo circuit elements. For example, two coupled elements may be directlyconnected by way of a metal line, or indirectly connected by way of anintervening circuit element (e.g., a capacitor, resistor, inductor, orby way of the source/drain terminals of a transistor). Variousmodifications to the preferred embodiment will be apparent to those withskill in the art, and the general principles defined herein may beapplied to other embodiments. Therefore, the present invention is notintended to be limited to the particular embodiments shown anddescribed, but is to be accorded the widest scope consistent with theprinciples and novel features herein disclosed.

FIG. 1 shows an exemplary robotic system 200 that is modified to includea multimodal sensing architecture according to an exemplary embodimentof the present invention. Robot system 200 generally includes a robotmechanism 201 and a control circuit 203, and multimodal sensingarchitecture 100 includes sensor arrays 101-1 and 101-2 disposed on anend effector 250 of robot mechanism 201. As described in detail below,sensor arrays 101-1 and 101-2 are operably coupled to control circuit203, and control circuit 203 is configured to control operationsperformed by robot mechanism 201 (including end effector 250) inresponse to sensor data generated by sensor arrays 101-1 and 101-2.

Referring to the upper portion of FIG. 1, robot mechanism 201 includesvarious mechanisms and structures that are operably configured inaccordance with known techniques to manipulate a target object 90 by wayof selectively actuating electrical motors. In the exemplary embodimentrobot mechanism 201 includes a shoulder/base mechanism 210 that isfixedly attached to a work surface (not shown) by way of a fixed base211, an upper arm structure 215 extending from the shoulder/basemechanism 210 to an elbow mechanism 220, a forearm structure 225extending from the elbow mechanism 220 to a wrist mechanism 230, a wriststructure 235 extending from the wrist mechanism 230 to hand/axialrotation mechanism 240, and an end effector 250 operably connected to aterminal portion of the hand/axial rotation mechanism 240. End effector250 is a hand/gripper-type mechanism having two gripper fingers 255-1and 255-2 that open (move away from each other) or close (move towardeach other) in accordance with the corresponding actuation of motorsmounted inside the gripper structure. Robot mechanism 201 also includesan optional camera 270 that is mounted near end effector 250 andprovides image data to controller 203. As mentioned above, robotmechanism 201 is merely introduced to provide a context for explainingthe features and benefits of multimodal sensing architecture 100, andthe specific configuration of robot mechanism 201 is not intended tolimit the appended claims.

Referring to the simplified block diagram located at the center rightportion of FIG. 1, control circuit 203 includes a control signalgenerator 205 that is configured to control operations performed byrobot mechanism 201 and end effector (hand) 250 in response to datareceived from several sources via wires (not shown) or othertransmission medium. As described in the background section (above)control signal generator 205 receives a programmed series of movementoperation/control instructions 207, and generates corresponding robotcontrol signals RMC that are transmitted via wires (not shown) tospecific electric motors disposed in robot mechanism 201, whereby targetobject 90 is manipulated in a programmed manner using end effector 250(e.g., robot mechanism is actuated to move end effector toward object90, and then end effector 250 is actuated such that gripper fingers 255press against opposite sides of target object 90). In addition toprogrammed control instructions 207, control circuit 203 also receivesimage-type feedback data 271 that may be used to adjust the programmedoperations in the manner described in the background section. However,as also explained in the background section and depicted in the upperleft portion of FIG. 1, a region 277 of the camera's vision field 275 istypically occluded by portions of end effector 277, which limits thefunctionality of image data 271. Moreover, as explained in thebackground section, single-modality sensors fail to provide theinformation needed to avoid many common industrial accidents and/or toperform complex assembly processes. Accordingly, control circuit 203 isdistinguished over conventional control circuits in that it alsoutilizes sensor-type feedback data generated by multimodal sensingarchitecture 100 in the manner described below.

Referring to FIGS. 1 and 2, multimodal sensing architecture 100 includessensor arrays 101-1 and 101-2 that are respectively fixedly attached toopposing contact surfaces 257-1 and 257-2 of gripper fingers 255-1 and255-2, where each sensor array 101-1 and 101-2 includes multiplesuperpixels 102 that are fixedly disposed over corresponding surfaceportions of contact surfaces 257-1 and 257-2 and operably connected toan associated readout circuit. For example, as indicated in the enlargedbubble region showing a tip portion 256 of gripper finger 255-1 in FIG.1 (and also in FIG. 2), sensor array 101-1 includes multiple superpixels 102, each disposed over a different portion (areal region) ofcontact surface 257. For example, superpixel 102-1 is disposed overcontact surface portion 257-11 of contact surface 257-1, and superpixel102-2 is disposed over contact surface portion 257-12 of contact surface257-1. As further depicted in FIG. 2, superpixel 102-21 of array 101-2is disposed over contact surface portion 257-21 of contact surface257-2, and superpixel 102-2 is disposed over contact surface portion257-22 of contact surface 257-2.

Each superpixel 102 includes multiple sensor nodes S that measure anassociated different stimuli applied to its corresponding contactsurface portion. For example, as indicated in the block diagram providedin the lower left portion of FIG. 1, superpixel 102-1 includes apressure sensor 103 configured to generate pressure (static event) dataPSD in response to an amount of static force SF received atcorresponding surface portion 257-1, a vibration sensor 104 configuredto generate vibration (dynamic event) data VD in response to mechanicalvibrations MV applied onto corresponding surface portion 257-1, anoptional proximity sensor 105 configured to generate proximity data PXDin response to a detected air-gap proximity distance PXD betweencorresponding surface portion 257-1 and an adjacent object (e.g., targetobject 90), and an optional temperature sensor 106 configured togenerate temperature data TD in response to a local temperature LTapplied to corresponding surface portion 257-1. In exemplaryembodiments, pressure sensor 103 of each superpixel 102 is implementedby a strain gauge, a capacitive pressure sensor, a piezoelectric sensoror a piezoresistive sensor, vibration sensor 104 of each superpixel 102is implemented by a piezoelectric sensor, a piezoresistive sensor, or amicromechanical system (MEMS) accelerometer, proximity sensor 105 ofeach superpixel 102 is implemented using a capacitive-coupling-typesensing element, and temperature sensor 106 of each superpixel 102 isimplemented using a resistive temperature detectors (RTD) or athermoelectric element.

Referring to FIG. 2, each array 101-1 and 101-2 also includes anassociated readout circuit 107-1 and 107-2 configured to receive sensordata from all of the array's superpixels and to transmit the receivedsensor data onto an associated shared sensor data bus for transmissionto control circuit 203. For example, as indicated in the bubble portionin FIG. 1, array 101-1 includes a first readout circuit portion 107-11that collects sensor data from the various sensors of superpixel 102-1,and a second readout circuit portion 107-12 that collects sensor datafrom the various sensors of superpixel 102-2, where both readoutportions 107-11 and 107-12 form part of readout circuit 107-1 (FIG. 1),which passes the collected sensor data onto shared sensor data bus108-1. As indicated in FIG. 2, the sensor data collected by readoutcircuit 107-1 includes pressure data PSD-1 and PSD-2 generated bypressure sensors 103-1 and 103-2 of superpixels 102-1 and 102-2, andvibration data VD-1 and VD-2 from vibration sensors 104-1 and 104-2.Similarly, as shown in FIG. 2, readout circuit 107-2 of array 101-2collects pressure data PSD-21 and PSD-22 generated by pressure sensors103-21 and 103-22 and vibration data VD-21 and VD-22 from vibrationsensors 104-21 and 104-22, and passes this collected data onto sharedbus line 108-2 for transmission to control circuit 203.

As indicated near the bottom of FIG. 1, “raw” sensor data PVPTtransmitted on shared sensor bus 108-1 may either be transmitteddirectly to control circuit 203 (i.e., along the single-dot-dashline/arrow), or transmitted indirectly to control circuit 203 by way ofan optional multimodal control generator circuit 109 (i.e., as indicatedby the double-dot-dash arrows). That is, in the case of thesingle-dot-dash-arrow, embodiment “raw” sensor data PVPT (i.e., seriallytransmitted pressure, vibration, proximity and temperature data) istransmitted directly to control signal generator 205. In this case,control signal generator 205 is modified to interpret sensor data PVPTand to generate appropriately responsive robot mechanism control signalsRMC. In the alternative (double-dot-dash) embodiment, the requiredmodification of control circuit 203 may be reduced by way of providingand configurating multimodal control generator circuit 109 topre-process sensor data PVPT, and to generate multimodal control signalsMCS that allow control signal generator 205 to generate theappropriately responsive robot mechanism control signals RMC withminimal processing time. For example, as indicated in FIG. 2, controlcircuit 203 is configured to control operations performed by endeffector 250 in response to said multimodal control signals MCS. by wayof transmitting robot mechanism control signals RMC(225-1) andRMC(225-2), which cause the end effector, e.g., to increase or decreaseforces FP1 and FP2 applied by gripper fingers 255-1 and 255-2 on anobject (not shown). In a practical embodiment, multimodal controlgenerator circuit 109 is implemented using an application-specificintegrated circuit (ASIC) or field programmable gate array (FPGA) thatis configured to generate appropriate output signals in response topredetermined patterns occurring in sensor data PVPT.

FIGS. 3A to 3D depict gripper fingers 255-1 and 255-2 of robotic system200 (shown in FIG. 1) during a series of operations involving graspingand moving target object 90, and illustrate an exemplary method forcontrolling a robotic system to move an object that is grasped in itsend effector. In particular, FIGS. 3A to 3D illustrate an example of howthe multimodal sensing architecture of the present invention isbeneficially utilized to enhance the ability of robotic systems toquickly identify a slipping-type non-standard operating condition and toautomatically implement an appropriate corrective operation by way ofincreasing the gripping force applied by end effector 250 on targetobject 90.

FIG. 3A depicts target object 90 at an initial time t0 when targetobject is positioned at a pre-designated start position, and the roboticsystem positions gripper fingers 255-1 and 255-2 (i.e., by way ofcausing fingers 255-1 and 255-2 on opposite sides of target object 90).Note that in FIG. 3A all sensor output is assumed to be zero (e.g.,PSD−1=0) because there is no contact between superpixels 102-1, 102-2,102-21 and 102-22 and the sides of target object 90.

FIG. 3B depicts target object 90 at a subsequent time t1 when targetobject is grasped between fingers 255-1 and 255-2, which is achieved byway of actuating appropriate mechanisms of the robotic system to movegripper fingers 255-1 and 255-2 toward each other such that grip forcesFP1(t 1) and FP2(t 1) are respectively applied by fingers 255-1 and255-2 on the sides of target object 90. At this point, pressure sensors103-1, 103-2, 103-21 and 103-22 are utilized to respectively generatestatic event data PSD-1, PSD-2, PSD-21 and PSD-22 in response to staticforces applied by target object 90 to corresponding contact surfaceportions 257-1 and 257-2 on end effector fingers 255-1 and 255-2,respectively. Note that the recorded force value “1” is arbitrarilyselected, and that vibration data values VD-1, VD-2, VD-21 and VD-22 areassumed to be zero to simplify the description.

FIG. 3C depicts target object 90 at a subsequent time t2 while targetobject is grasped between fingers 255-1 and 255-2 and being transportedby the robotic system from the pre-designated start position to apre-designated destination position. For descriptive purposes it isassumed that target object 90 undergoes a slipping (dynamic) event inwhich object 90 slips an amount −Z relative to fingers 255-1 and 255-2at some point during transport. Note that the grip forces FP1(t 1) andFP2(t 1) respectively applied by fingers 255-1 and 255-2 on the sides oftarget object 90 have not changed from time t1, so all static event dataPSD-1, PSD-2, PSD-21 and PSD-22 remains unchanged (i.e., equal to “1”);that is, pressure sensors 103-1, 103-2, 103-21 and 103-22 are not ableto detect the slipping event. However, slipping events of this typegenerate characteristic mechanical vibrations (forces) MV1 in fingers255-1 and 255-2 that are detectable by vibration sensors 104-1, 104-2,104-21 and 104-22. According to an aspect of the present invention,vibration sensors 104-1, 104-2, 104-21 and 104-22 are utilized toquickly identify the slipping event by way of respectively generatingnon-zero dynamic event data VD-1, VD-2, VD-21 and VD-22 in response tomechanical vibrations MV1. By configuring control circuit 203 toproperly interpret the static and dynamic sensor data (e.g., byidentifying that a slipping event is occurring when all static eventdata remains unchanged and all dynamic event data increases uniformlyduring transport of an object), the control circuit 203 is able toquickly implement a corrective action (i.e., adjust either the robotmechanism or the end effector) in response to the identified slipping(non-standard operating) condition, thereby preventing further slippingand possible loss of target object 90. For example, as indicated in FIG.3D, a suitable corrective action may involve actuating appropriatemechanisms of the robotic system to move gripper fingers 255-1 and 255-2toward each other such that grip forces FP1(t 3) and FP2(t 3) applied byfingers 255-1 and 255-2 on the sides of target object 90 are higher thanthose applied at time t2. Note that successful application of thecorrective action is also immediately detected by way of an expectedincrease in static event data (e.g., pressure data values PSD-1, PSD-2,PSD-21 and PSD-22 increase from “1” at time t3 to “2” at time t4) and aconcomitant decrease in dynamic event data (e.g., vibration data valuesVD-1, VD-2, VD-21 and VD-22 decreased from “1” at time t3 to “0” at timet4).

FIGS. 4A to 4D depict gripper fingers 255-1 and 255-2 of robotic system200 (shown in FIG. 1) during a series of operations involving mounting aprimary object 91 (e.g., a hollow cylinder with an open bottom end) overa secondary object 92 (e.g., a solid cylinder), and illustrate anexemplary method for controlling a robotic system during a relativelycomplex assembly process. In particular, FIGS. 4A to 4D illustrateexamples of how multimodal sensing architecture of the present inventionmay be beneficially utilized to enhance the ability of robotic systemsto adjust to various misalignments that may prevent completion of theassembly process if performed using conventional methods.

FIG. 4A depicts a time t0 when primary object 91 is grasped betweenfingers 255-1 and 255-2 and moved into a pre-designated position formounting over secondary object 92. Similar to the situation describedabove with reference to FIG. 3B, pressure sensors 103-1, 103-2, 103-21and 103-22 are utilized to respectively generate static event dataPSD-1, PSD-2, PSD-21 and PSD-22 in response to static forces applied byprimary object 91, that the indicated pressure force value “1” isarbitrarily selected, and that vibration data values VD-1, VD-2, VD-21and VD-22 are assumed to be zero for brevity.

FIG. 4B indicates a first misalignment event occurring a time t1. Inthis case primary object 91 is displaced by a small distance −X1relative to secondary object 92, whereby a lower right edge portion ofprimary object 91 contacts an upper surface of secondary object 92 asthe robotic mechanism lowers primary object 91 in the −Z direction.Because the grip forces applied by fingers 255-1 and 255-2 on the sidesof primary object 91 have not changed from time t0 to time t1, allstatic event data PSD-1, PSD-2, PSD-21 and PSD-22 remains unchanged(i.e., equal to “1”). However, the impact between objects 91 and 92generates mechanical vibrations (forces) MV2 that radiate throughfingers 255-1 and 255-2 in a characteristic manner such that the pointof impact may be determined by combining static event data with thedynamic event data collected by vibration sensors 104-1, 104-2, 104-21and 104-22. For example, when the static event data remains constant andvibration data VD-22 from vibration sensor 104-22 is higher (e.g., “4”)than vibration data VD-21 from vibration sensor 104-21 (e.g., “3”), andboth are higher than vibration data from vibration sensors 104-1 and104-2, then an impact-type dynamic event may be identified and an impactlocation may be estimated. With this information, the robotic system'scontrol circuit is able to automatically perform a corrective adjustment(e.g., by moving primary object 91 a small amount in the X direction),thereby achieving a suitable alignment between primary object 91 andsecondary object 92 for the mounting process to continue.

FIG. 4C indicates a second misalignment event occurring a time t2. Inthis case primary object 91 is positioned adequately to facilitatemounting over secondary object 92, but a minor displacement relative tosecondary object 92 results in scraping (rubbing) contact between aportion of primary object 91 and secondary object 92. As in the previousexample, static event data PSD-1 and PSD-2 remains unchanged (i.e.,equal to “1”), but the scraping-type contact between objects 91 and 92generates characteristic mechanical vibrations (forces) MV3 that allowthe vibration sensors to identify the location of the scraping-typecontact. Additionally, the imbalance in forces from the lack of contactof object 92 and 255-1, but contact with object 92 and object 255-2, maycreate a detectible pressure change by PSD-21 and PSD-22. For example,when the static event data remains constant and vibration data VD-21 andVD-22 is higher (e.g., “2”) than vibration data VD-1 and VD-2 (e.g.,“1”) during the assembly operation, then a scraping-type dynamic eventis occurring and contact point is near finger 255-2. With thisinformation, the robotic system's control circuit is able toautomatically perform a corrective adjustment (e.g., by moving primaryobject 91 a small amount in the X direction), thereby achieving anoptimal alignment between primary object 91 and secondary object 92 thatfacilitates a scraping-free mounting process.

FIG. 4D depicts primary object 91 and secondary object 92 at thecompletion of the mounting process, which occurs at a time t3 whenprimary object 91 has been fully lowered over secondary object 92. Inaddition to utilizing the multimodal sensing architecture to detectimpact-type or scraping-type contact for purposes of taking correctiveaction, the combination of dynamic event data and static event data mayalso be utilized to confirm the successful completion of an assemblyoperation by way of recording an expected final contact (or non-contact)event. For example, the successful mounting of primary object 91 onsecondary object 92 may produce characteristic mechanical vibrations MV4only when precise alignment between the objects has been achieved.Conversely, mechanical vibrations MV4 may only be generated when theassembly process was completed incorrectly. In either case, the abilityto detect both static and dynamic event data allows the multimodalsensing architecture to provide information that cannot be obtainedusing cameras or single-modality sensors.

Although the examples of FIGS. 3A to 4D are described with reference tosuperpixels that include two separate sensor (i.e., a pressure sensor103-x and a vibration sensor 104-x), the methodology utilized in theseexamples may be implemented using a single multimodal sensor in place ofeach superpixel, provided each single multimodal sensor is capable ofdetecting both static and dynamic events, and provided the readoutcircuit is configured to separately generate both static event data anddynamic event data from an output signal generated by each multimodalsensor. In alternative single-multimodal-sensor embodiments eachsuperpixel of the array described above is replaced with either apiezoelectric sensor or a piezoresistive sensor; both of these sensortypes qualify as multimodal sensors in that they generate sensor outputsignals including both static event characteristics (e.g., directcurrent magnitude) and dynamic event characteristics (e.g., alternatingcurrent magnitude). By configuring the readout circuits to separate andmeasure the static/dynamic characteristics in the output signalgenerated by each multimodal sensor (e.g., by way of filters and otherknown signal processing techniques), the static/dynamic event datavalues described above with reference to FIGS. 3A to 4B are madeavailable for use by a robotic system's control circuit, therebyfacilitating implementation of the associated automatically performedcorrective adjustments.

FIG. 5 is a simplified cross-sectional side view showing a sensor array101A of a multimodal sensing architecture according to a specificembodiment of the present invention. Data readout from sensor networksor arrays of sensor nodes are often realized by using wires orelectrical interconnects directly routed to the sensor nodes. However,though simplistic and convenient, these wires become cumbersome whendealing with a large number sensor network with thousands of nodes.Specifically, for application such as tactile sensing/exploration inrobotic end effectors, this type of wiring, with interconnects directlyin the sensor plane, becomes a real issue. With a large array ofmultimodal sensor nodes, this configuration is not only impractical(both in terms of footprint and data acquisition), but can alsointroduces noise and cross-talk in the robotic tactile sensors. Thetopology implemented in sensor array 101A addresses this problem by wayof providing a frontplane sensor layer 110A and a separate backplanereadout layer 120A that are integrally connected in a way that minimizeswiring between the large number of sensors in sensor array 101A and ahost robotic system control circuit. In one embodiment, the varioussensors of each superpixel (e.g., pressure sensor 103A and vibrationsensor 104A) are fabricated or otherwise disposed on sensor layer 110A,and layout circuit 107A is disposed on readout layer 120A, wherepressure sensor 103A and vibration sensor 104A are operably coupled toreadout circuit 107A by way of via-type signal lines 115A-1 and 115A-2,respectively, that extend between sensor layer 110A and readout layer120A.

In one embodiment, the topology of sensor 101A further includes one orboth of a protective layer/matrix 130A disposed over sensor layer 110A,and a base substrate 140A disposed under readout layer 120A. Inpractical embodiments, protective layer/matrix 130A comprises one of aflexible material (e.g., silicone), silicon or a hard shell material(e.g., aluminum, where appropriate such as around the perimeter), andhas a thickness in the range of one micron to one millimeter. In otherembodiments, optional base substrate 140A comprises one of silicon,glass, steel, plastic and aluminum, and has a thickness in the range often microns and one millimeter.

FIG. 6 is an exploded perspective view showing an exemplary sensor array101B according to another specific embodiment. Sensor array 101B has atopology similar to that of array 101A, including a sensor layer 110B, areadout layer 120B, an optional protective layer 130B and a basesubstrate 140B. In this case, sensor layer 110B includes a siliconsubstrate 111B on which superpixels 102 are fabricated using CMOS ormicrofabrication techniques such that each superpixel includes thevarious sensor types described above (e.g., superpixel 102B-1 includes apressure sensor 103B, a vibration sensor 104B, a proximity sensor 105Band a temperature sensor 106B. In addition, readout layer 120B alsoincludes a silicon substrate 121B with readout circuit 107B implementedthereon by way of CMOS fabrication techniques such that each superpixel102B on substrate 101B is aligned with a corresponding readout circuitportion 124B (e.g., such that sensor nodes 103B to 106B of superpixel102B-1 are aligned with input nodes provided in corresponding readoutcircuit portion 124B-1). During the manufacturing process, substrate111B is fixedly attached to substrate 121B by a die-attach method suchthat pressure sensor 103B and vibration sensor 104B are operably coupledto corresponding input nodes 125B-1 and 125B-2 of readout circuit 107Bby way of bumps 116B-1 and 116B-2 (e.g., indium bumps, solder bumps orpolymer bumps), which respectively form at least a portion of signalpaths 115B-1 and 115B-2 extending between substrates 111B and 121B.

FIG. 7 is a simplified cross-sectional side view showing a sensor array101C having a topology according to another specific embodiment. In thiscase, each superpixel 102C-1 and 102C-2 of sensor array 101C isrespectively fabricated on a separate associated silicon island 114C-1and 114C-2, and silicon islands 114C-1 and 114C-2 are mounted over areadout layer 120C (e.g., by way of an intervening silicon substrate111C). The various sensors of superpixels 102C-1 and 102C-2 are operablycoupled to associated readout circuit portions 107C-1 and 107C-2 by wayof via-type signal lines 115C, which are produced using any of therelevant techniques mentioned herein.

FIG. 8 is a block diagram depicting a readout circuit 107D of a sensorarray according to another specific embodiment. Readout circuit 107Dincludes an array of readout portions (pixels) RCP-00 TO RCP-24 that arerespectively configured and operably coupled to receive associatedsensor data (e.g., pressure data values PSD-00 to PSD-24, respectively)from corresponding pressure sensors (not shown). Readout circuit 107Dalso includes readout control circuits (e.g., a row select circuit 810,a column select circuit 820) that are configured to sequentiallytransfer the sensor data (e.g., pressure data values PSD-00 to PSD-24)from each readout portion RCP-00 TO RCP-24 to a digital readout circuit830, which in turn is configured to convert the sensor data into digitalvalues for transfer to a control circuit or other circuit.

FIG. 9 is a block diagram depicting a readout circuit portion 107Eaccording to another specific embodiment. Readout circuit portion 107Ecomprises analog front-end 910 including an analog-to-digital converter(ADC) 913 and a digital-analog-converter (DAC) 915, digital circuitry920 including digital logic and signal processing, and a digitalcommunications interface 930. Analog circuitry 910 is configured toreceive analog sensor data (e.g., pressure data value PSD-00A) fromcorresponding sensor nodes of associate superpixels (e.g., sensor 103E),and configured to generate digital data values (e.g., digital pressuredata value PSD-00D) that is transmitted to digital circuitry 920.Digital circuitry 920 is configured to transfer the digital sensor datato a host robotic system's control circuit via the communicationsinterface 930.

FIG. 10 is a simplified cross-sectional side view showing a sensor array101F according to another specific embodiment. Data acquisition fromsensor network or arrays of sensor nodes, especially on flexiblesubstrates, are often realized by the use of wires or electricalinterconnects directly routed to the sensor nodes. The interconnects onpolymer substrates typically lie on the same plane. While this approachsimplifies the manufacturing process, these wires become cumbersome whendealing with a large number sensor network with thousands of nodes.Specifically, for application such as tactile sensing/exploration inrobotic end effectors, this type of wiring, with interconnects directlyin the sensor plane, becomes a real issue. With a large array ofmultimodal sensor nodes, this configuration is not only impractical(both in terms of footprint and data acquisition) but can alsointroduces noise and cross-talk in the robotic tactile sensors. Rigidsilicon backplanes can also be impractical in some instances becausethey can break with any kind of bending, flexing or twisting.Accordingly, sensor array 101F includes a flexible electronics inspiredtopology with an amorphous silicon backplane integrated to a frontplanesensor architecture to address this problem.

Referring to FIG. 10, array 101F includes a sensor layer 110F having oneor more sensors (e.g., pressure sensor 103F) formed on a first substrate111F, and a readout layer 120F including second substrate 121F on whicha readout circuit 107F is fabricated as an amorphous-silicon (a-Si)integrated circuit including a plurality of a-Si thin-film transistor(TFT) elements. In one embodiment, one or more sensor nodes (e.g.,vibration sensor 104F) is/are also fabricated on readout layer 120Fusing TFT elements. In one embodiment custom a-Si TFT readout circuit107F is a pixel matrix with addressable pixels.

Although the present invention has been described with respect tocertain specific embodiments, it will be clear to those skilled in theart that the inventive features of the present invention are applicableto other embodiments as well, all of which are intended to fall withinthe scope of the present invention. For example, although the presentinvention is described with specific reference to articulated-typerobotic systems that use two-finger end effectors, the multimodalsensing architecture disclosed herein may also be beneficially utilizedin advanced robotic systems that utilize three, four or five finger endeffectors (e.g., human-like robotic hands), and may also be utilized inother mechanical systems as well, such as on prosthetic limbs. In someembodiments a multimodal sensing architecture may comprise a singlesensor array operably mounted on a single finger of a multi-finger endeffector or on a probe-like end effector. Further, those skilled in theart will understand that the sensors and superpixels of the presentinvention can have different interrelated configurations, orientations,placement of nodes, sensors, array size, periodicity/aperiodicity,different circuit configurations inside the pixel array, etc., whilestill embodying the spirit and scope of the inventive concept. Forexample, although the invention is described primarily with reference toCMOS or TFT-type sensors fabricated on associated silicon substrates,suitable sensor and superpixel arrays may also be produced by forming orplacing sensors/superpixels on printed circuit boards (PCBs) usingwell-known PCB fabrication techniques.

The invention claimed is:
 1. A method for controlling a robotic systemincluding a robot mechanism having an end effector including a pluralityof spatially arrayed superpixels, each said superpixel including anassociated pressure sensor and an associated vibration sensor disposedover a corresponding contact surface portion of the end effector, theend effector being configured to grasp a target object, and the robotmechanism being configured to move the target object while being graspedby the end effector, the method comprising: controlling the pressuresensors of said plurality of spatially arrayed superpixels such thateach said pressure sensor generates associated time-based localizedstatic event data values in response to static forces applied to saidcorresponding contact surface portion on the end effector while thetarget object is grasped by the end effector; controlling the vibrationsensors of said plurality of spatially arrayed superpixels such thatsaid each said vibration sensor generates associated time-basedlocalized dynamic event data values in response to vibrational forcesapplied to said corresponding contact surface portion on the endeffector while the robot mechanism is moving the target object; whilethe robot mechanism is moving the target object, identifying a locationof a non-standard operating condition using changes in at least one ofsaid time-based localized static event data values and said time-basedlocalized dynamic event data values; and adjusting the end effector inresponse to said identified non-standard operating condition location.2. The method of claim 1, further comprising: using both said staticevent data and said dynamic event data to identify slipping of saidtarget object relative to said end actuator, and conrolling the endeffector to increase a grip force applied to the target object inresponse to said identified slipping.
 3. The method of claim 1, whereinidentifying said non-standard operating condition location includesusing both said static event data and said dynamic event data toidentify an undesirable contact location at which undesirable contactbetween said target object and a secondary object occurs, and whereinadjusting the end effector comprises controlling the robot mechanism tomove the target object to a location that relative to the secondaryobject such that the undesirable contact is avoided.
 4. The method ofclaim 1, wherein controlling the robotic system comprises performing anassembly process including the target object and a secondary object. 5.The method of claim 4, wherein the target object comprises a hollowstructure having an open bottom end, and wherein performing the assemblyprocess comprises mounting the target object over the secondary objectsuch that at least a portion of the secondary object extends through theopen bottom end into the hollow structure when the assembly process issuccessfully completed.
 6. The method of claim 4, wherein identifyingthe location of the non-standard operating condition comprises detectinga misalignment between said target object and said secondary object bydetecting changes in one or more time-based localized dynamic event datavalues generated by associated vibration sensors located adjacent to animpact location between said target object and said secondary objectduring said assembly process.
 7. The method of claim 4, whereinidentifying the location of the non-standard operating conditioncomprises detecting a misalignment between said target object and saidsecondary object by detecting changes in one or more time-basedlocalized dynamic event data values generated by associated vibrationsensors located adjacent to a scraping-type contact location betweensaid target object and said secondary object during said assemblyprocess.
 8. The method of claim 4, further comprising using both saidstatic event data and said dynamic event data to detect an expectedfinal contact event that confirms a successful completion of saidassembly process.